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Abstract 

Math Snacks animations and support materials were developed for use on the web and mobile technologies to teach 
ratio, proportion, scale factor, and number line concepts using a multi-modal approach. Included in Math Snacks are: 
Animations which promote the visualization of a concept image; written lessons which provide cognitive complexity 
for understanding; and active, situated learning activities to facilitate memorable experiences to deepen 
comprehension. This pilot study compared pre-post test gains for 460 sixth and seventh grade students enrolled in 
nine different classrooms. In five of the nine classrooms, teachers utilized the Teacher Guide that corresponded with 
the five Math Snacks animations and one game and in four classrooms teachers used the same Math Snacks 
animations and one game, but were free to develop their own lessons using available online resources. Results 
showed moderate and significant pre-post test gains for all six grade students. However, significant gains for seventh 
grade students were shown only for classrooms where the teacher Guide was used. While it appears that the use of 
Teacher Guide is useful only for seventh grade, such a conclusion is premature given the small number of classrooms 
and the exploratory nature of this investigation. Further analyses of moderator variables (e.g., instructional fidelity, 
learner characteristics) are certainly necessary. 

Keywords: educational games; mathematics teaching and learning; computer animations 


1. Introduction 

1.1 The Role of Technology’ in Teaching 

As laptops and tablets become more accessible to teachers and students in the mathematics classroom, animations 
and games may play a greater role in instruction. Therefore, it is important to develop effective technological tools 
that are designed to support student learning and academic achievement. Current technological pedagogy has shown 
that the effective use of technology, particularly constructivist-oriented technology, may lead to greater student 
achievement (Wenglinsky, 1998; Schacter, 1999; Lei, 2007). However, technology alone does not necessarily lead to 
knowledge transfer of mathematics skills and understanding (Devlin, 2011, Gee, 2007; Pea, 1987; Prensky, 2011). 
Wenglinsky (1998) found evidence that using technology can support mathematics learning but contends that 
constructivist applications had a more positive effect on math achievement than traditional uses of drill and practice 
software. Additionally, research has shown that teaching students mathematics conceptually rather than 
algorithmically leads to a deeper understanding of the content (Resnick, 1983; Kilpatrick, 2001; Ogbuehi, 2007). 
This deep conceptual understanding of elementary and middle school mathematics content is crucial for students to 
be successful in high school mathematics and beyond (Kilpatrick, 2001). However, due to the large amount of 
content covered in ‘middle school mathematics’ in the U.S. curriculum, it can be difficult to determine which 
particular concepts students struggle with (Schmidt, Houang, & Cogan, 2002). 

1.2 Gaps in Mathematics Knowledge 

In 2005, researchers evaluated the scores of standards based assessments for over 24,000 children from five diverse 
school districts in New Mexico. The analysis showed key gaps in mathematical learning for middle school students 
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in the areas of: ratio, proportion, scale factor, fractions, number line, and place value (Wiburg, Korn & Villa, 2013). 
Wiburg, et al. triangulated this analysis with teachers’ common mathematics misconceptions as shown on the 
Mathematics Knowledge for Teaching measure, (Hill, Schiling, & Ball, 2004) and over 800 hours of observation of 
mathematics classrooms to identify key areas of need for middle-school mathematics instruction. They found that 
teachers showed knowledge gaps in the same areas (fractions, ratios, number system concepts, etc.). 

How can we address key areas of need for middle school mathematics instruction? The Math Snacks grant was 
written to address these specific learning gaps. The Math Snacks project is a five-year grant funded by the National 
Science Foundation (award number 0918794) for the development and testing of animations, games, and curricular 
support materials designed to increase students’ understanding of middle school mathematics concepts in the areas of 
ratio, proportion, scale factor, fractions, number line, and place value. These materials were developed consistent 
with current theory related to multimedia learning (Mayer, 2002; Moreno, 2006) and constructivist instruction 
(Greenes, 2009). The materials underwent three years of continuous development which included informal learner 
observations, student focus groups, and comments from teachers and students. 

1.3 Math Snacks Instruction 

Math Snacks are short animations and mini-games designed to present mathematics utilizing web-based technology. 
The short animations and games were designed to encourage student learning of mathematics content. 
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Figure 1: Descriptions of Math Snacks, Five Animations and One Game 
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Figure 1 describes the five short animations and one game used in this study. Each Math Snack has a corresponding 
lesson protocol that is designed to guide the teachers’ use of the animations and games. These protocols contain a 
guided discussion for the animation, an active situated learning activity to be completed after the animation; and 
suggested questions to facilitate deeper comprehension of the lesson. The lesson protocols as well as the learner 
guides, teacher guides, example lesson videos and other support materials are available online at 
http://mathsnacks.com/teachingBD.html 

1.4 Purpose 

One of the goals of the Math Snack Grant was to determine the effectiveness of the Math Snacks curriculum on 
mathematics achievement. To accomplish this goal, the grant will conduct a large-scale randomized controlled 
efficacy study in fall 2013 and spring 2014. In preparation for the large-scale study, the Math Snacks research team 
conducted this pilot study in order to achieve the following objectives: 

• Determine the psychometric properties of the Measure of Mathematics Learning. 

• Determine the short-term effects of the Math Snacks games and animations 

• Determine whether the use of the corresponding teacher guides add to the overall effects of the Math Snacks 
games and animations. 

• Collect qualitative data describing teachers’ classroom implementation of the math snacks materials. 

1.5 Research Questions 

• Research Question One: Will students who were taught using the Math Snacks games and animations show 
growth in mathematical knowledge in the targeted areas of math achievement (i.e., number-line & ratio/ 
proportion)? 

• Research Question Two: Is student math achievement influenced by teacher’s use of the teacher guides that were 
developed by the Math Snacks grant to support math instruction? 


2. Method 

2.1 Participants 

Teacher participants were recruited from one urban and one rural southwestern border-school district. Nine teachers 
from two school districts (7 urban & 2 rural) volunteered to participate in this study. Teachers were given the Math 
Snacks instructional materials (games, animations, and teacher guides) as well as minimal monetary incentives for 
their participation in the study. Student participants included 460 students (236 male, 224 female). Some posttest 
participant attrition occurred for twenty-six sixth grade students (18 control group students, 8 experimental group 
students) but there was no posttest attrition for seventh grade students. This study was approved by the respective 
school districts and by the New Mexico State University Institutional Review Board. Parental permission was 
obtained for all student participants. 

2.2 Procedure 

Students received instruction for eight weeks. In order to examine the effects use of the corresponding teacher guides, 
five of the nine teachers were asked to follow Math Snacks Teacher Guide (experimental group) and the remaining 
four teachers were encouraged to develop their own lessons (self-constructed lesson or control group). Classrooms 
were systematically assigned to each condition. The Math Measure was administered prior to and then following the 
8-week instructional period. Teachers in both groups were asked to integrate five animations and one game during 
the 8-week period. The researchers observed each classroom on five occasions during the 8-week instructional period. 
Student learning was operationalized as pre-post test gain score. To investigate whether student learning differed 
across the two conditions (experimental and control), teachers were systematically assigned to each condition. 

2.3 Reliability and Validly of the Measure of Mathematics Learning 

While student mathematics performance is measured annually using a standardized measure, the purpose of this 
investigation was to determine more discrete change in mathematics knowledge over time. Therefore, the Math 
Snacks research team constructed a measure, the Measure of Mathematics Learning, which was intended to measure 
short-term progress in discrete areas of mathematical knowledge (number line and ratio/proportion). 

2.3.1 Score Reliability of the Measure of Mathematics Learning 

Score reliability was determined using coefficient alpha. Alpha was calculated for the pre-test sample at .88 for 6th 


Published by Sciedu Press 


156 


ISSN 1927-2677 E-ISSN1927-2685 




www.sciedu.ca/jct 


Journal of Curriculum and Teaching 


Vol. 2, No. 2; 2013 


grade (n = 138) and at .83 for 7th grade (n = 150) students. Alpha values above .80 are considered good and values 
above .90 are considered excellent (George & Mallory, 2003; Nunnally & Bernstein, 1994). 

2.3.2 Score Validity of the Measure of Mathematics Learning 

The twenty-seven item measure was constructed from released items that were publically available from national test 
databases. Three members of the research team located released mathematics test-items from a variety of sources that 
included the National Assessment of Educational Progress (NAEP), the Massachusetts Comprehensive Assessment 
System (MCAS), the Florida Comprehensive Assessment Test (FCAT) and the California Standards Test (CST). 

Table 1: Measure of Mathematics Learning Test Blueprint 



Multiple-choice 

Open-ended 

Total 

Number Line 

1,2,3,4,5 

5a,6a,6b,6c,6oe 

10 

Ratio/Proportion 

7,8,10,11,12,13, 

14,16,17,18 

9a,9aexp,9b,9bexp, 

15a,15aoe,15b 

17 

Total 

15 

12 

27 


The test blueprint for the Measure of Mathematics Learning is shown in Table 1. In order to assure appropriate 
coverage for grades six and seven, the test items were selected from test banks that were intended for grades five 
through eight. As Table 1 shows, the items included recognition (i.e., multiple-choice), as well as inference and 
explanation (i.e., open-ended) items. Twenty-seven items were selected. Ten items pertaining to number line 
concepts (5 multiple-choice, 5 open-ended) and seventeen items pertaining to ratio/proportion concepts (10 
multiple-choice, 7 open-ended) were selected. 

2.4 Qualitative Observations 

Classroom observations were conducted to determine how the classroom teachers carried out their mathematics 
lessons. Each teacher was observed three times by trained observers who judged how well teachers carried out six 
key instructional elements. Observers were trained to assign a summary categorical judgment (yes/sometimes/no) to 
estimate how often each of the six key instructional elements occurred during a mathematics lesson. If the observers 
determined that the key element was present over 75% of the time, the categorical judgment was considered as 
“YES”. If the observers determined that the action took place between 50% and 74% of the time, the categorical 
judgment was considered as “SOME”. If the observer determined that the action took place less than 50% of the time, 
the categorical judgment was considered as” NO”. The key instructional elements included: 

• Animation: The teachers’ use of the animation to support the lesson. 

• Learner Guide Use: The teachers’ utilization of the written learner guide (separate support document from the 
Teacher Guide) to support his/her lesson. 

• Activity: How frequently the teacher encouraged an active learning experience where students could gain a 
deeper understanding of the content. 

• Student Engagement: How often students appeared attentive and engaged during the lesson (i.e., asking 
questions, discussing the content with their classmates, etc.). 

• Effective Questioning: How frequently the teacher asked “open-ended” questions that encouraged student 
inquiry in the lesson rather than “closed-ended” questions that simply requested factual information. 

• Effective Classroom Discourse: How frequently the teacher engaged in classroom discourse that encouraged 
students to interact with the technology, with the lesson, with each other (pertinent to the lesson), or with the 
teacher for the purpose of learning the lesson content. 

The lead researcher reviewed the transcripts of the trained observers and compiled a summary that described how 
each teacher applied the six key instructional elements. The lead researcher met with the trained observers weekly to 
recalibrate observational judgments. 
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3. Results 

3.1 Descriptive Analyses 

Sixth and seventh grade test scores were aggregated by grade level for both the control and experimental groups. 

Table 2: Means, Standard Deviations, and Sample Size for Sixth and Seventh Grade Experimental and Control 
Groups 


Grade Level 

Control 

Pretest 

Posttest 

Pretest 

Experimental 

Posttest 

Sixth 

10.46 

12.93 

10.73 

13.21 


(5.48) 

(5.69) 

(5.20) 

(5.62) 


n = 146 

n = 146 

G\ 

OO 

II 

G 

3 

II 

OO 

SO 

Seventh 

14.27 

15.12 

12.50 

14.74 


(5.30) 

(6.03) 

(4.97) 

(6.24) 


n =67 

n =67 

n = 58 

OO 

ITT 

II 

G 


As Table 2 shows, both control and experimental groups showed pre/posttest gains on the Measure of Mathematics 
Leaning. While the mean pre/posttest gain for sixth grade appeared comparable for control and experimental groups 
(mean difference = 2.47 and 2.48 respectively), the pre/posttest gain for the seventh grade appeared much greater for 
the experimental group in comparison to the control group (mean difference = 2.24 and .85 respectively). 

3.2 Pre-Treatment Equivalency 

As Table 2 shows, pretest scores were comparable between experimental and control for both sixth and seventh grade. 
Follow up independent t-tests showed no significant differences in pretest scores between control and experimental 
groups, t (359) = .14, p = .89, and t (123) = 1.92, p = .06, for sixth and seventh grades respectively suggesting that 
the groups were equivalent on the dependent measure prior to the intervention. 

3.3 Research Questions 

3.3.1 Research Question One: Will students who were taught using the Math Snacks games and animations show 
growth in mathematical knowledge in the targeted areas of math achievement (i.e., number-line &ratio/ proportion) 

3.3.2 Research Question Two: Is student math achievement influenced by teacher’s use of the Teacher Guides that 
were developed by Math Snacks grant to support math instruction. 

3.4 Statistics and Data Analysis 

To investigate both research questions, a mixed two-way repeated measures analysis of variance (ANOVA) was 
conducted with group (experimental & control) as the between-subject factor and time (pre/posttest) as the 
within-subject factor. Alpha was set at .05 for all analyses. 

For sixth grade students, there was a significant main effect for time, F (1, 333) = 131.39, p < .001, partial q 2 = .28. 
However, the main effect for group and the group by time interaction was not significant. This finding suggests that 
both the experimental and control groups made significant and parallel gains over time in learning the targeted 
mathematics concepts. For seventh grade students, there was a significant group by time interaction F (1, 123) = 4.09, 
p = .045, partial q 2 = .03. Therefore, interpretations of the main effects were set aside and the nature of the 
interaction was further investigated with follow-up dependent t-tests. The dependent t-test was not significant for the 
control group, t (66) = 1.86, p = .067. However, the experimental group showed a large and significant gain on the 
dependent measure over time, t (57) = 4.33, p < .001, d= .81. 

In summary, it appeared that sixth grade students made large and significant gains in mathematics knowledge over 
the course of the inteivention regardless of whether teachers used the Math Snacks Teacher Guide or not. However, 
for seventh grade students, significant and large gains were only shown for those students whose teacher used the 
Teacher Guide. 

3.5 Treatment Fidelity’ 

One of the stated purposes of this study was to collect qualitative data describing teachers’ classroom implementation 
of the math snacks materials. 
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Table 3: Classroom Observation Summaries 


Key Elements 


Animation 

Learner 

Activity 

Student 

Effective 

Effective 

Teacher 


Guide 


Engagement 

Questioning 

Classroom 

(grade) 


Use 




Discourse 


Control Group (Teachers did not use 

corresponding Math SnacksTeacher Guide) 

B (6 th ) 

Yes 

Yes 

Some 

Some 

No 

No 

D (6 th ) 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

F (6 th ) 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

G (7 th ) 

Yes 

Yes 

Some 

No 

No 

No 


Experimental Group (Teachers did use corresponding Math SnacksTeacher Guide) 

A (6 th ) 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

C (6 th ) 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

E (6 th ) 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

H (6 th ) 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

I (7 th ) 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 


Note: Amount of time Key Element was present - Yes > 75%, Some 50 - 74%, No < 50% 

Table 3 summarizes the qualitative observational data for each classroom. As Table 3 shows, all teachers in the 
experimental group consistently demonstrated all six of the key instructional elements while teaching the 
mathematics lesson. However, in the control group, two teachers (one sixth and one seventh grade) were less 
consistent at demonstrating all of the key instructional elements during their classroom instruction. Teacher B (sixth 
grade) used the animation and the learner guide, but did not carry out the activities for all of the lessons. This teacher 
also struggled to maintain student engagement and did not show any evidence of effective questioning strategies or 
effective classroom discourse. Teacher G (seventh grade) had similar issues and also did not carry out all of the 
recommended activities. This teacher also struggled with classroom management. While students appeared engaged 
during the animation, they quickly lost interest in any further instruction and showed little engagement during the 
suggested learner guide activities. Subsequently, this teacher did not show evidence of effective questioning or 
classroom discourse. 


4. Discussion 

Instructional innovations offer the promise of improving instructional effectiveness but do not guarantee a beneficial 
instructional outcome. As Moreno (2005, p. 14) pointed out, “any technological innovation is fraught with promises 
and challenges”. It therefore becomes critical to carefully investigate the strengths and weaknesses of any 
instructional innovation. Math Snacks animations, games and curricular materials were developed to address key 
gaps in mathematical learning. The purpose of this investigation was to develop and pilot the dependent measure 
(Measure of Mathematics Learning - number line ratio & proportion), to determine if the Math Snacks materials 
effectively address certain key areas (e.g., number line and proportion) and to explore whether teachers benefited 
from the “extra level of guidance” offered when using the Math SnacksTeacher Guides. 

The 27-item Measure of Mathematics Learning showed good internal structure reliability and reasonable face 
validity. The success of this measure was largely due to the careful selection of test items from nationally recognized 
test banks, the utilization of a test blueprint in order to assure adequate construct coverage, and the review and 
modification of the test items from the five-member research team. This measure was used to determine student 
progress in the targeted mathematics areas (number line & proportion/ratio). 

In terms of program effectiveness, pre-post test scores on the dependent measure showed mixed results. Sixth grade 
students, regardless of whether teachers used the Teacher Guide, showed large and significant improvement on the 
dependent measure. However, significant improvement on the dependent measure was only shown for seventh grade 
students whose teacher used the Teacher Guide. On the surface, it appears that the use of the Math Snacks materials 
does result in significant growth in the targeted mathematics areas (i.e., ratio and proportion). In addition, for seventh 
grade, the use of the Math Snacks Teacher Guide appeared to result in a more favorable learning outcome. It could 
be that the Teacher Guide offerred an added level of guidance that enabled the teacher to more effectively structure 
and carryout the mathematics lesson. An alternate explanation however, is that the difference in seventh grade 
outcomes was due to teacher characteristics. In any case, this finding points out the necessity to include classroom 
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observational data in any study that attempts to explore the efficacy of an untested instructional program. 

In addition, further investigation should study the complex relationship between game-play and mathematical 
knowledge acquisition. As Gee (2011) points out, our current task is to present logical and testable hypotheses in 
order to move our understanding of the benefits, limitations, and possible harmful effects of game-based instruction. 
Further areas of inquiry should include learning context. Both Fe (2008) and Plass et al. (2011) found an increase in 
affective variables depending upon whether students engaged in individual, cooperative, or competitive game play. 
However, the question remains whether this increase in affect results in a corresponding increase in mathematical 
knowledge. 

Given the exploratory nature of this pilot study certain limitations should be considered. Due to the small number of 
classrooms that were included in this study, random selection to condition would not likely have achieved 
pre-treatment group equivalency. While treatment and control groups showed equivalent scores on the pretest, they 
may have differed in other areas. Future studies should include a larger sample of classrooms and use either 
matching or random assignment in order to assure pre-treatment group equivalency. In addition, these findings are 
limited to sixth and seventh-grade students receiving an eight-week focused instruction in the area of number line 
and ratio/proportion. These findings may not generalize to other grade levels or other mathematics topics. 
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